The present disclosure relates to cloud computing and database organization and, more specifically, to a method and system for dynamic optimization of data aggregation.
Enterprises process and analyze data on a daily basis. Such data may be located in objects such as tables, which are stored on data sources. Data sources may include databases and non-database data sources. Users may easily access and process data within objects stored on databases. Specifically, users may use instructions developed using database languages such as Structured Query Language (“SQL”) to select and process specific data within objects stored on a database.
However, problems may arise when users are working with data that is stored on non-database data sources. Such data sources may not support SQL or other types of database languages. Thus, selected data may have to be moved from those data sources to a virtual database in order to be processed using conventional database tools. When the amount of selected data is large, this may result in huge inefficiencies. In particular, a lot of memory may be required to store the selected data in the virtual database, significant time may be spent moving the selected data to the virtual database, and high network traffic due to the transfer of selected data may cause other operations to slow down.